This specification generally relates to using neural networks for object recognition. Some neural networks are recurrent neural networks that include one or more Long Short-Term Memory (LSTM) layers. Some LSTM layers are convolutional LSTM layers.